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INSTITUTE  GOALS: 

The  goals  of  the  Social  Science  Research  Institute  are  threefold: 

•  To  provide  an  environment  in  which  scientists  may  pursue  their 
own  interests  in  some  blend  of  basic  and  methodological  re¬ 
search  in  the  investigation  of  major  social  problems. 

•  To  provide  an  environment  in  which  graduate  students  may  receive 
training  in  research  theory,  design  and  methodology  through 
active  participation  with  senior  researchers  in  ongoing  research 
projects. 

•  To  disseminate  information  to  relevant  public  and  social  agencies 
in  order  to  provide  decision  makers  with  the  tools  and  ideas 
necessary  to  the  formulation  of  public  social  policy. 

HISTORY: 

The  Social  Science  Research  Institute,  University  of  Southern  California, 
was  established  in  1972,  with  a  staff  of  six.  In  fiscal  year  1978-79,  it 
had  a  staff  of  over  90  full-  and  part-time  researchers  and  support  person¬ 
nel.  SSRI  draws  upon  most  University  academic  Departments  and  Schools  to 
make  up  its  research  staff,  e.g.  Industrial  and  Systems  Engineering,  the 
Law  School,  Psychology,  Public  Administration,  Safety  and  Systems  Manage¬ 
ment,  and  others.  Senior  researchers  have  joint  appointments  and  most 
actively  combine  research  with  teaching. 

FUNDING: 

SSRI  Reports  directly  to  the  Executive  Vice  President  of  USC.  It  is 
provided  with  modest  annual  basic  support  for  administration,  operations, 
and  program  development.  The  major  sources  of  funding  support  are  federal, 
state,  and  local  funding  agencies  and  private  foundations  and  organizations. 
The  list  of  sponsors  has  recently  expanded  to  include  governments  outside 
the  United  States.  Total  funding  has  increased  from  approximately  $150,000 
in  1972  to  almost  $3,000,000  in  the  fiscal  year  1978-1979. 

RESEARCH  INTERESTS: 


Each  senior  SSRI  scientist  is  encouraged  to  pursue  his  or  her  own  research 
interests,  subject  to  availability  of  funding.  These  interests  are  diverse; 
a  recent  count  identified  27.  Four  major  interests  persist  among  groups 
of  SSRI  researchers:  crime  control  and  criminal  justice,  methods  of  dispute 
resolution  and  alternatives  to  the  courts,  use  of  administration  records 
for  demographic  and  other  research  purposes,  and  exploitation  of  appli¬ 
cations  of  decision  analysis  to  public  decision  making  and  program  eval¬ 
uation.  But  many  SSRI  projects  do  not  fall  Into  these  categories.  Most 
projects  combine  the  skills  of  several  scientists,  often  from  different 
disciplines.  As  SSRI  research  personnel  change,  its  interests  will  change 
also. 
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I.  Introduction 


This  final  report  summarizes  the  work  by  the  Social  Science 
Research  Institute,  University  of  Southern  California,  on  con¬ 
tract  N'o.  \00014-79-C-0038  from  the  Advanced  Research  Projects 
Agency,  Department  of  Defense.  The  research  conducted  during 
this  contract  period  from  December  15,  1978  to  June  30,  1980, 
under  the  direction  of  Professor  Ward  Edwards,  the  Principal 
Investigator,  grew  out  of  a  program  of  research  supported  by 
ARPA  for  the  study  of  decision  making  and  the  decision  sciences. 
Edwards  (1973,  1 97 5) , Edwards  and  Seaver  (1976)  ,  and  Edwards,  John, 
and  Stillwell  (1977,  1979)  summarize  previous  research. 

The  proposal  leading  to  this  contract  called  for  research  on 
three  specific  topics:  Validation  of  multiattribute  utility 
elicitation  techniques,  simplification  of  assessment  procedures 
for  multiattribute  elicitation,  and  error  in  utility  judgment. 
Our  research  on  these  and  other  topics  is  contained  in  six 
technical  reports  which  have  been  produced  or  are  now  being 
prepared.  Summaries  of  these  technical  reports  appear  in  this 
report . 

This  report,  in  addition  to  providing  the  summaries  of  our 
technical  reports,  attempts  to  examine  how  the  three  topics 
fit  together,  what  the  findings  we  have  made  with  respect  to 
them  mean  for  decision  technology,  and  where  to  go  from  here. 
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II.  Why  One  Should  Care  About  Validation, 

Error,  and  Simplification 

Application  of  the  ideas  and  procedures  of  decision  analysis 
in  DOD  and  other  contexts  has  been  expanding  at  a  phenomenal 
rate,  while  training  of  qualified  analysts  has  proceeded  much 
more  slowly.  We  expect  both  trends  to  continue.  The  combi¬ 
nation  of  exponentially  increasing  demand  and  linearly  in¬ 
creasing  supply  has  already  created  a  problem  that  we  expect 
to  become  much  worse:  How  can  we  make  decision  analytic  aid 
widely  available  without  sacrificing  standards  of  quality? 

If  training  more  analysts  is  an  adequate  answer  only  in  a 
too-long  run,  we  need  another.  A  second  answer,  which  has 
motivated  much  of  our  work  for  the  last  two  or  three  years 
is  decision  analysis  without  the  analyst. 

What  must  we  do  to  make  decision  analysis  without  the 
analyst  feasible  and  effective?  First,  we  must  simplify  both 
models  and  elicitation  procedures,  without  undue  sacrifice 
of  validity  of  either.  In  many  situations,  decision  makers 
can  perform  analyses  of  sufficient  complexity  for  a  good 
decision  with  minimal - to -no  resort  to  an  analyst--if  the 
methods  are  easy  enough  to  understand  and  use.  Our  vali¬ 
dation  work  grows  in  part  from  this  goal.  We  investigated 
elicitation  techniques  that  cover  the  range  from  theoretically 
sophisticated,  judgmentallv  complex  procedures  to  techniques 
that  call  for  little  or  even  no  elicitation.  We  were  seeking 
an  answer  to  the  question,  "how  simple  can  the  elicitation 
be  and  still  capture,  not  only  the  judgments  of  the  respondent 
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but  also  his  or  her  specific  expertise?"  To  foreshadow  one 
of  our  main  conclusions,  the  answer  is  "very  simply  indeed!" 

In  order  to  ask  this  question  sensibly,  we  were  forced 
not  only  to  develop  simple  methods,  but  to  satisfy  ourselves 
that  they  worked.  This  step,  much  neglected  in  the  history 
of  the  development  of  decision  analytic  methods,  is  not 
satisfied  by  use  of  such  criteria  as  client  satisfaction  or 
even  convergent  validation  with  holistic  judgments.  It 
specifically  requires  criterion  validation- -which  is  what  we 
used. 


Error  is  important  because  (a)  it  puts  an  upper  bound  on 
criterion  validity,  and  (b)  it  is  likely  to  vary  among  eli¬ 
citation  methods.  If  a  theoretically  sophisticated  elici¬ 
tation  method  leads  to  more  error,  random  or  systematic,  than 
a  less  elegant  one,  the  less  elegant  one  may  well  be  preferable 
for  that  reason  alone. 

Yet  another  approach  to  performing  decision  analysis  without 
the  analyst  consists  of  developing  prototypical  decision 
structures,  applicable  to  whole  classes  of  decisions.  Though 
we  regard  this  as  a  highly  promising  strategy,  we  have  done 
no  research  relevant  to  it. 

III.  Experimental  and  Theoretical  Studies 


III. A.  A  comparision  of 


derived  from  holistic 


rank  order  judgments'!" 


importance  weights  for  MAUA 
indifference,  direct  subjective, 


and 


College  student  subjects  were  taught  a  four  attribute  MAU 
model  of  diamond  worth  using  the  paradigm  of  multiple  cue 
probability  learning  and  outcome  feedback.  After  training, 
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subjects  assessed  MAU  weight  parameters  via  a  variety  of  eli¬ 
citation  techniques.  Three  task  variables  were  manipulated: 
monetary  payoff  versus  no  payoff,  task  uncertainty  (II  versus 
181  error  variance  in  criterion),  and  number  of  learning  trails. 

Results  showed  little  difference  between  levels  for  any 
of  the  task  variables  in  terms  of  either  weight  parameters 
or  composite  matching (derived  from  consistent  application  of 
the  weights  used  to  generate  outcome  feedback  compared  with 
application  of  judgmental  weight  sets).  Significant  differences 
were  found  between  elicitation  procedures  for  the  mean  number 
of  inversions  in  rank  ordering  of  attributes  with  bootstrapping 
(weight  parameters  derived  via  regression  from  holistic  judg¬ 
ment  of  profiles)  providing  the  least  inversions.  However, 
despite  this  difference  between  weight  judgments,  comparisons 
of  composite  worth  derived  from  these  weight  sets  showed  that 
all  elicitation  procedures  did  well,  with  minimal  differences 
between  them.  A  number  of  rank  weighting  procedures  resulted 
in  equally  high  quality  composite  worth  measures. 

III.B.  Evaluating  credit  applications:  A  validation  of  multi¬ 
attribute  utility  techniques  against  a  real  world  criterion. 


Credit  loan  officers  from  a  California  bank  served  as 
subjects  in  a  criterion  validation  of  multiattribute  utility 
elicitation  techniques.  The  criterion  against  which  the  judg¬ 
ments  were  compared  was  a  statistically  derived  credit  scoring 
model  which  the  bank  uses  to  determine  creditworthiness  of 
applicants  for  credit  cards  and  small  unsecured  loans. 


mmmm 
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Results  of  this  experiment  are  very  similar  to  those  from 
the  MCPL  experiment  discussed  above.  Weight  parameters  showed 
some  difference  between  techinques  with  holistic  bootstrapping 
and  constant  sum  point  distribution  procedures  providing 
weight  sets  that  corresponded  most  closely  to  weights  from  the 
bank  model.  From  a  practical  standpoint,  however,  three  indices 
of  decisions  produced  by  those  weight  sets  showed  that  all 
decomposition  procedures  produced  high  quality  decisions  while 
the  holistic  procedure  produced  somewhat  inferior  results. 

Again,  as  in  the  MCPL  experiment,  several  rank  order  weighting 
procedures  performed  at  the  high  level  of  the  other,  more 
difficult  assessment  procedures. 

III.C.  Reference  effects:  A  sheep  in  wolf's  clothing. 

In  many  of  its  forms  multiattribute  utility  elicitation 
depends  on  comparison  of  holistic  profiles  to  derive  weights 
(or,  as  they  are  sometimes  called,  scaling  constants)  for 
attributes  in  decision  problems.  The  typical  judgment  called 
for  is  a  difference  between  two  profiles,  the  first  with  a 
given  attribute  at  its  best  level,  the  second  with  that 
same  attribute  at  its  worst  level,  and  in  each  case  the 
rest  of  the  attributes  3t  some  common  level  for  each  of 
the  two  profiles.  A  recent  paper  fYates  and  Jagacinski, 

1979)  argues  that  these  holistic  judgments  may  be  subject 
to  systematic  bias  due  to  the  values  given  to  the  irrelevant 
They  also  report  data  which,  they  suggest,  shows 


variable  s. 
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In  response  to  this  paper,  Barron  and  John  (1980)  show 
that  the  findings  are  easily  accounted  for  by  a  multiplicative 
MAU  model  rather  than  the  hypothesized  judgmental  bias.  They 
also  exhibit  sufficient  conditions  for  two  other  predictions, 
i.e.,  no  effect  and  the  opposite  effect.  Finally,  Barron  and 
John  present  multiattribute  evaluation  data  exhibiting  these 
latter  two  effects. 

III.D.  Equal  weights,  flat  maxima  and  trivial  decisions. 

This  study  differentiates  between  the  evaluation  of  deci¬ 
sions  and  evaluating  decision  rules  in  mul t i at t r i but e  utility 
decision  making.  The  authors,  John  and  Edwards,  show  that  in 
fact,  most  decisions  are  relatively  trivial  and  anv  decision 
model  that  incorporates  the  correct  attributes,  properly 
oriented  (more  is  better  than  less  or  less  is  better  than  more 
will  result  in  high  quality  decisions  when  compared  to  an  op¬ 
timal  decision  rule.  The  authors  go  on  to  show  that  this  "hig 
quality"  of  decisions  is  a  direct  result  of  the  set  of  options 
that  are  being  considered.  When  a  large  set  is  included, 
with  many  that  would  not  be  chosen  using  any  decision  rule 
(sure  losers),  and  many  that  are  obvious  choices  under  any 
rule  (sure  winners),  of  course,  all  rules  will  seem  to  do  well 
An  interesting  decision  problem  is  the  correct  selection  from 
among  those  in  the  set  of  contenders,  i.e.,  the  options 
remaining  after  sure  winners  and  sure  losers  are  removed  from 
the  option  set.  Thus,  for  evaluation  of  decision  rules  the 
nature  of  the  contender  set  must  be  known. 
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In  a  Monte  Carlo  simulation  study  the  authors  examined 
some  typical  decision  problems,  to  find  the  expected  numbers 
of  sure  winners,  sure  losers  and  general  contenders  resulting 
from  varying  model  choice  (additive  versus  no  restrictions) , 
expected  correlations  between  pairs  of  attributes  (either 
.2,  0,  .5,  or  .9),  proportion  of  alternatives  to  be  chosen 
(less  than  1%  to  501),  number  of  dimensions  (2,  3,  4,  5),  and 
initial  number  of  alternatives  (50  or  100).  They  found  that 
as  attributes  become  more  highly  positively  correlated,  the 
percentage  of  sure  winners  and  losses  increases  dramatically 
while  the  number  and  percentage  of  contenders  decreases. 

Higher  numbers  of  dimensions  leads  to  more  contenders  and  fewer 
sure  winners  and  losers.  Finally,  the  additive  utility  res¬ 
triction  leads  to  quite  modest  increases  in  sure  winners  where¬ 
as  the  proportion  of  sure  winners  seem  to  increase  roughly 
linearly  with  the  proportion  to  be  chosen.  Comparable  statements 
can  be  made  about  sure  losers. 

The  most  important  finding  is  that,  regardless  of  the 
correlation  among  dimensions  in  the  original  set,  the  correlation 
of  dimensions  in  the  contender  set  will  be  negative- -with 
size  dependent  on  number  of  dimensions.  This  is  expected, 
since  sure  winners  and  sure  losers  achieve  that  status  by  doing 
well  or  badly  on  all  dimensions. 

The  result  presents  an  interesting  dilemma  in  evaluating 
decisions,  rules,  approximations,  etc.  The  methods  by  which 
options  enter  an  option  set  are  often  ill -specif ied  and  seldom 
systematic.  Yet  most  techniques  for  evaluating  decision  rules 


depend  on  how  well  they  do  within  the  original  option  set. 
It  is,  for  example,  consideration  of  the  entire  option  set 
that  leads  to  the  effectiveness  of  equal  weights. 


Since  any  decision  rules  should  be  capable  of  identifying 
sure  winners  and  sure  losers,  a  strong  case  supports  the  view 
that  decision  rules  and  procedures  should  be  evaluated  according 
to  their  performance  on  contenders  only.  Such  a  case  leads 
to  two  conclus ions , (1 )  none  of  the  past  conclusions  about  ro¬ 
bustness  of  decision  procedures  in  the  presence  of  various 
kinds  of  error  are  as  sure  as  we  thought  they  were,  but  (2) 
the  amount  of  utility  loss  produced  by  a  suboptimal  decision 
rule  is  relatively  small  if  measured  over  the  whole  option 
set,  though  often  much  larger  if  measured  over  the  contender 
set  only.  Choice  of  which  standard  to  use  in  evaluating  a 
decision  rule  seems  to  be  a  relevant  philosophical  question. 


III.E.  Functional  relationships  between  risky  and  riskless 
multiattribute  utility  functions. 

Expected  utility  theory  and  conjoint  measurement  theory 
form  two  major  classes  of  models  and  assessment  procedures  to 
construct  multiattribute  utility  functions.  In  conjoint 
measurement  theory  a  value  function  v  is  constructed  which 
preserved  preferences  among  riskless  mult iattributed  outcomes. 
The  risky  utility  function  constructed  in  the  framework  of 

expected  utility  theory,  also  preserves  such  riskless  pre- 
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ferences.  In  addition  it  is  an  appropriate  guide  for  decisions 
under  uncertainty  since  its  expectation  preserves  risky  pre¬ 
ference  among  gambles.  Since  both  u  and  v  are  order  preserving 
functions,  they  must  be  related  by  a  strictly  increasing 
transformation.  However,  u  and  v  need  not  coincide  or  be  re¬ 
lated  through  any  special  functional  forms,  unless  some  simple 
decomposition  forms  are  assumed.  More  restricted  functional 
relationships  obtain,  if  u  and  v  are  assumed  to  be  either 
additive  or  multiplicative.  In  particular,  u  can  be  shown  to 
be  linearily,  logarithmically,  or  exponentially  related  to  v, 
depending  on  which  function  is  additive  and  which  is  multipli¬ 
cative.  A  recent  paper  by  v. Winterfeldt  (1980)  proves  such 
functional  relationshps  based  on  the  theory  of  functional 
equations.  Techniques  are  described  to  assess  the  parameters 
of  these  functions. 


III.F,  Rank  weighting  in  multiattribute  utility  decision 
making:  avoiding  the  pitfalls  of  equal  weights! 


IVainer,  1976  discusses  a  number  of  conditions  that,  once 
they  hold,  assure  high  quality  prediction  from  equally  weighted 
prediction  attributes.  A  study  by  Stillwell  and  Edwards  (1979), 
takes  the  opposite  tack,  describing  a  number  of  conditions, 
commonly  found  in  decision,  rather  than  prediction,  situations 
that  could  lead  to  large  losses  in  utility  from  the  resulting 
decisions  if  equal  weighting  of  evaluative  dimensions  were  used. 
The  study  goes  on  to  suggest  that  rank  weighting  of  evaluative 
dimensions  provides  a  remedy  for  these  situations  and  at  the 
same  time  retains  much  of  the  most  appealing  characteristic  of 
equal  weighting,  simplicity  of  elicitation. 
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To  make  the  point  the  study  reanalyzes  the  data  from 
three  multiattribute  utility  decision  making  studies,  com¬ 
paring  the  results  of  several  rank  weighting  procedures, 
as  well  as  equal  weights,  to  those  from  the  judges  in  the 
original  elicitations.  It  was  found  that  equal  weights  led 
to  large  decrements  in  the  correlation  of  composites  when 
compared  to  rank  weighting  of  attributes.  In  general, 
the  rank  weighting  techniques  provided  remarkably  good  approx¬ 
imations  to  the  assumed  "true"  weights  provided  by  the  judges. 
Within  the  conditions  of  this  study  and  for  the  purposes  of 
multiattribute  decision  making,  rank  weights  seem  to  be  suf¬ 
ficient  improvements  over  equal  weights  to  warrant  the  extra 
effort  involved  in  their  elicitation. 

IV.  Some  Implications 

In  the  aggregate,  the  findings  of  these  studies  begin  to 
suggest  approaches  to  implementing  the  slogan  "decision 
analysis  without  a  decision  analyst." 

The  first  point  is  that,  at  least  as  far  as  weighting  is 
concerned,  we  apparently  can  trust  the  results  obtained  from 
research  on  college  students  to  generalize  effectively  to 
results  obtained  from  highly  trained  experts  working  in  the 
topics  of  their  expertise.  Moreover,  it  seems  to  make  little 
difference  which  elicitation  technique  for  weights  is  used, 
so  long  as  that  technique  incorporates  at  least  rank-order 
information.  Equal  weights  are  clearly  inferior,  and  holistic 
judgments  used  as  a  basis  for  inferring  weights  have  special 
problems  of  their  own.  But  virtually  any  form  of  rank  weighting 
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procedure  is  likely  to  do  as  well  as  the  most  sophisticated 
Keeney-Raiffa  elicitation  techniques.  Rank  weighting  can  be 
done  easily,  and  does  not  require  the  intervention  of  a  decision 
analyst . 

The  second  point  is  that,  at  least  for  monotonic  physical 
dimensions,  shapes  of  single-dimension  utility  functions  make 
little  if  any  difference.  It  follows  that  straight  lines  will 
be  admirable  proxies  for  any  monotonic  utility  function,  for 
all  purposes  other  than  those  to  which  issues  of  risk  aversion 
are  directly  relevant.  We  would  in  fact  argue  that  they  work 
well  in  contexts  in  which  attitude  toward  risk  is  important 
also,  but  the  data  and  thinking  of  this  year’s  program  do  not 
bear  on  the  argument. 

The  third  point  is  that  the  nature  of  the  aggregation  rule, 
from  the  limited  set  in  common  use  by  decision  analysts,  is 
sufficiently  unimportant  that  one  might  as  well  settle  for  the 
convenient  and  simple  weighted  linear  rule. 

What  does  matter,  then,  if  all  these  things  do  not?  First 
and  foremost,  of  course,  is  the  structuring  of  the  evaluative 
and/or  decision  problem  itself.  Whether  a  decision  analysis 
is  simple  or  complex  makes  little  difference  unless  it  deals 
with  the  correct  options,  the  correct  dimensions  of  the  multi¬ 
attribute  utility  function,  and  the  real  decision  makers  and 
other  stakeholders.  None  of  this  year's  work  bears  on  the 
important,  and  virtually  unaddressed,  problem  of  simplifying 
and  validating  the  task  of  structuring  decision  problems. 
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Some  decision  problems  come  with  relatively  simple 
structures.  This  is  especially  true  of  problems  that  have 
a  generic  character.  An  obvious  example  in  military  environ¬ 
ments  is  source  selection  for  hardward  procurement.  Although 
the  hardware,  the  potential  sources,  and  the  relevant  dimensions 
of  value  will  vary  from  instance  to  instance,  such  problems 
can  be  expected  to  have  a  generic  structure.  Aspects  of  the 
required  performance  will  have  been  prespecified  by  the  govern¬ 
ment;  other  aspects  will  be  specified  explicitly  or  implicitly 
by  the  sources.  The  sources  are  the  options,  and  they  will 
identify  themselves.  Cost  variables  are  likely  to  arise;  the 
decision  about  whether  additional  capability  justifies  addi¬ 
tional  cost  will  almost  always  need  examination.  And  issues 
concerning  the  dependability,  history  of  past  cost  overruns, 
history  of  past  failures  to  deliver  on  time,  and  related 
questions  about  the  sources  will  normally  be  relevant.  It  is 
by  no  means  absurd  to  imagine  that  over  time  a  generic  model 
of  source  selection  decisions  might  be  developed,  and  that 
it  could  be  combined  with  the  simple  elicitation  techniques 
already  discussed  to  permit  its  use  without  decision  analyst 
assistance . 

At  a  much  lower  level,  a  similar  kind  of  argument  applies 
to  many  personnel  decisions.  The  options  are  often  well 
known.  The  relevant  dimensions  are  specified  by  the  job 
description,  plus  whatever  others  may  be  relevant  but  not 
included  in  that  document.  Both  original  appointments  and 
promotions  could  be  so  handled. 
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The  idea  of  generic  structures  has  already  been  applied 
to  siting  decisions,  with  considerable  success,  by  Keeney.  Our 
argument  is  simply  that  for  decisions  for  which,  generic  struc¬ 
tures  can  be  developed,  the  goal  of  decision  analysis  without 
the  analyst  is  attainable  now,  based  on  what  we  already  know. 

But,  unfortunately,  most  decisions  do  not  lend  themselves 
to  generic  structuring.  Consequently,  an  obvious  next  step 
toward  that  goal  is  to  think  harder  than  we  have  been  about 
how  to  facilitate  the  structuring  process  without  the  aid 
of  an  analyst. 

The  preceding  discussion  has  assumed  the  validity  and 
generalizability  of  our  finding  that  decision-analytic 
elicitations  meet  the  requirement  of  criterion  validity. 

While  our  data  are  certainly  encouraging,  they  come  from  only 
one  study.  Moreover,  that  study  is  concerned  only  with  weights. 
Worst  of  all,  the  experts  in  that  study  were  experts  not  only 
about  the  problem,  but  also  about  the  criterion.  A  defensible 
interpretation  of  our  results  (which  we  do  not  believe)  would 
be  that  we  in  effect  asked  our  respondents  to  feed  the  cri¬ 
terion  back  to  us,  and,  knowing  it,  they  did  so. 

The  obvious  solution  to  that  problem  is  to  conduct  experi¬ 
ments  in  a  context  in  which  (a)  subjects  are  expert,  (b)  a 
criterion  is  available,  and  fc)  the  subjects  know  nothing 
of  the  criterion.  Requirements  a  and  c  seem  so  nearly  to 
contradict  each  other  that  such  an  experiment  seems  a  priori 
hard  to  imagine.  Nevertheless,  it  could  be  done  if  the  criterion 
were  developed  from  an  appropriate  data  base  as  a  part  of  the 


PAGE  14 


validation  study  itself.  Circumstances  permitting  that  do 
exist . 

Having  studied  only  one  kind  of  expert,  we  are  in  no 
position  to  generalize  to  other  experts  or  other  kinds  of 
expertise.  That  limitation  is  inherent  in  every  empirical 
study,  and  we  do  not  apologise  for  it.  Indeed,  in  view  of 
the  difficulty  of  finding  available  experts,  we  are  pleased 
to  have  been  able  to  study  criterion  validity  with  expert 
judges  at  all.  It  is  distressing,  though  unavoidable,  that 
the  problem  of  what  an  expert  is,  how  one  recognizes  him  or 
her,  and  how  expertise  on  a  subject  matter  relate  to  expertise 
in  decision  technology  in  performing  decision  analytic  tasks 
remains  virtually  terra  incognita  12  years  after  development  of 
the  basic  ideas  that  make  such  questions  crucial.  But  these 
are  among  the  toughest  questions  decision  analysts  must  either 
face  or  assume  away;  it  is  not  surprising  that  with  few  ex¬ 
ceptions  they  have  done  the  latter. 

As  decision  analysis  develops,  our  present  sophisticated 
but  unsystematic  and  unscientific  knowledge  d>out  such  questions 
will  come  to  be  replaced  with  real  studies  of  them--but  only 
gradually,  as  those  controlling  the  time  of  experts  come  to 
be  sufficiently  impressed  with  the  importance  of  such  questions 
to  be  willing  to  invest  that  time  in  the  research  that  can 
answer  them  properly. 
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